Using video information to control cursor position

ABSTRACT

A cursor control system for a processor-based system may receive input commands through a digital camera. Displacements of the camera along two transverse axes may correlate to displacements of a cursor image on a display screen associated with the processor-based system. Thus, the camera may be utilized for image capture operations and likewise for controlling the processor-based system&#39;s cursor image. The camera may then function not only as a conventional camera but also as a pointing device.

BACKGROUND

This invention relates generally to processor-based systems andparticularly to systems for controlling the cursor position on displayson such systems.

The cursor is a graphical user interface which indicates a focus for aprocessor-based system. One common cursor image is an arrow image whichmay be moved across a computer screen to select different items thatprovide input signals for software running on the system. The cursor mayalso be implemented by a technique in which text or other indicia arehighlighted and the position of the highlighting may be moved. In eithercase, a particular item may be selected, usually by operating apushbutton when the cursor image is associated with a particularselectable item.

Cursors are generally positioned using pointing devices. Conventionalpointing devices include a mouse, a touchpad, a keypad, a roller ball ora touch screen.

A variety of computers send and receive streaming video. For example,processor-based systems may implement video conferencing using atethered digital camera. In addition, cameras may be utilized forimplementing Internet telephone calls as another example. Thus, it iscommon that a camera may remain operational while the user uses thesystem for other functions which may not involve video.

Despite the fact that there are a number of ways to provide cursorcontrol signals and a variety of different pointing devices, there iscontinued interest in pointing devices which provide added advantages.One problem with existing cursor control or pointing devices is thatthey tend to be, to one degree or another, counterintuitive. To somedegree motion of the pointing device, such as a mouse, does notcorrespond to the motion of the cursor on the screen. Thus, it would bedesirable to link the motion in a more intuitive way with the motion ofthe cursor on the screen.

Another problem with many conventional pointing devices such as themouse is that they are difficult to use in a confined space. For examplein an airplane, relatively little room is available for the mouse. Themouse does not work well if it cannot be moved in a translating motionto a sufficient extent. Thus, it would be desirable to have better waysfor operating pointing devices that work in confined spaces.

Thus, there is a continuing need for better ways to provide cursorcontrols especially in systems which use video cameras.

SUMMARY

In accordance with one aspect, a method of controlling a processor-basedsystem includes receiving video information from a camera coupled to thesystem. Changes in the orientation of the camera are translated withcursor control signals.

Other aspects are set forth in the accompanying detailed description andclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front elevational view of one embodiment of the presentinvention;

FIG. 2 is an enlarged perspective view of the digital camera shown inFIG. 1;

FIG. 3 is a flow chart for software for implementing one embodiment ofthe present invention;

FIG. 4 is a flow chart for software which may be used to implement oneof the features illustrated in the software shown in FIG. 3, inaccordance with one embodiment of the present invention;

FIG. 5 is a schematic depiction of the pixel samples which may beutilized to determine the direction of camera movement in accordancewith one embodiment of the present invention;

FIG. 6 is a flow chart for software which may be utilized in connectionwith the software illustrated in FIG. 3 in one embodiment of the presentinvention; and

FIG. 7 is a block diagram for the system shown in FIG. 1.

DETAILED DESCRIPTION

A processor-based system 10, shown in FIG. 1, may include a processorhousing 12, a keyboard 13 and a display 14. In addition, a digitalcamera 16 may be coupled to the system 10 by a cable 28. While aconventional desktop computer is shown, the present invention isapplicable to processor-based systems in general, including laptopcomputers, processor-based appliances and set top computers asadditional examples.

Turning next to FIG. 2, the digital camera 16, in accordance with oneembodiment of the present invention, includes a housing 18 containing acamera lens 24. The housing 18 is supported for rotation about a base26. In addition, the housing 18 may tilt up and down relative to thebase 26. Thus, the housing 18 and the lens 24 may be redirected left orright and up and down.

The housing 18 also includes a focus adjustment knob 22. A pushbutton 20may be used for a variety of purposes including generating an inputcommand to capture a still image.

A wide variety of digital cameras may be utilized in connection with thepresent invention including those that use complementary metal oxidesemiconductor (CMOS) imaging arrays as well as those that use chargecoupled device (CCD) imaging arrays. The image captured by the camera 16may be transferred over the cable 28 to the processor-based system 10.However, in other embodiments, a processor may be included within thecamera itself and in such cases, the camera may include its ownprocessor-based system.

Cursor control software 30, stored on the system 10, operates when acursor input command is received (diamond 40). The software 30 checksfor camera displacement as indicated in block 32 of FIG. 3. Thedisplacement of the camera may be provided by the user to create acursor control signal in one of at least two ways. The user can simplyphysically pick up the camera and turn it left or right or up or down,as examples, to cause cursor displacement to be recognized by theprocessor-based system. This provides a very intuitive way of providingcursor commands. In addition it is amenable to use in relativelyconfined spaces.

Alternatively, the cursor commands can be implemented by displacing thecamera while it is positioned on a surface such as a desk. In this way,the camera housing 18 may be pivoted relative to the base 26. Again, awide range of motion is possible and in this alternative, the cursorcommands can be implemented by displacing the camera while it ispositioned on a surface such as a desk.

A check at diamond 34 determines whether camera motion has beendetected. If so, the direction of the motion is determined as indicatedin block 36. A cursor image 15 (FIG. 1) is moved in accordance with thedetermined direction of camera motion, as indicated in block 38.

If a mouse click is received as detected in diamond 40, the mouse clickmessage is redirected to a default operating system handler to implementa mouse button select feature (block 42). The mouse click may be anindication to select an item associated with the cursor or initiateother action in software. The mouse click signal may be generated inresponse to operation of the pushbutton 20 in one embodiment of thepresent invention.

Thus, the camera 16 may operate as a conventional mouse wherein the usercan simply turn and tilt the housing 18 with respect to the base 26thereby changing the direction which the camera lens 24 points. Thischange in orientation may be detected and a direction of motion may bedetermined. This information can be translated into cursor controlsignals. That is, a turn to the left may cause the mouse cursor to moveto the left and so on. The extent of motion may determine the extent ofmovement of the cursor. Likewise, when the cursor is associated with aselectable item on the display screen 14, operation of the pushbutton 20works similarly to the operation of a conventional mouse button, causingthe item to be selected as an input command to software operating on thesystem 10.

Referring next to FIG. 4, one embodiment of the software 44 fordetecting motion begins by grabbing a video frame as indicated in block46. A pixel sample set is selected for detecting motion. This is forprocessing convenience. That is, by selecting a pixel sample set foranalysis, the analysis is simplified since every pixel in the camera'simaging array need not be separately analyzed.

A variety of techniques may be utilized to determine the sample set. Inone embodiment of the present invention, the sample set may be arectangular array of pixels which is generally centrally disposed on theimaging array and which produces a centrally located rectangular portionof the image displayed on the display screen. Even within that sampleset, instead of taking every single pixel, a limited number of pixelsmay be analyzed in one embodiment of the invention. For example, in oneembodiment of the invention, every fourth pixel may actually be analyzedas part of the sample set.

As indicated in block 50, each pixel within the sample set may besuccessively selected. As indicated in block 52, a corresponding pixelin an old or previous frame and a new frame are accessed. Theircorresponding values are then subtracted as indicated in block 54. Thevalues, depending on the model used, can represent RGB color informationor a number of other color spaces including the YUV color space, the HSVcolor space, the YIQ color space, and the YDrDb color space, asadditional examples, which have color values which may be utilized inthe fashion described herein. Thus, in an example which uses a red,green, blue (RGB) color model, the red values in the old frame aresubtracted from the red values in the new frame and so on. The absolutevalue of the differences is computed as indicated in block 56.

If the result exceeds a noise threshold (as indicated in diamond 58),the result may be stored as indicated in block 60. In some embodimentsof the present invention, the noise threshold may be set by the user. Ifthe result does not exceed the noise threshold, the flow continues tostep through ensuing pixels in the sample set. When no more pixels areleft to be analyzed (diamond 62), the result is returned as indicated inblock 64. Particularly, as indicated in diamond 34 in FIG. 3, if motionis detected, the flow continues to determine its direction. Thus, aresult determined by the flow shown in FIG. 4 may be returned to theflow illustrated in FIG. 3.

Once it is known that the camera 16 has been displaced, the next step isto determine the direction of displacement of the camera. In someembodiments of the present invention, this may be done relativelyroughly since the direction may be repeatedly determined in very quicksuccession. For example, in one embodiment of the present invention, thedirection may be recalculated with each frame. Many digital camerasoperate at 30 frames per second. Thus, the system need only make arelatively rough estimate of the direction and if the estimate is wrong,it will be corrected with ensuing frames.

Thus, referring to FIG. 5, the initial sample set A for the first frame66 is analyzed. The software determines which of the sets B through Imost closely corresponds to the image which was previously recorded asthe set A. In this way, the system can create a vector for the directionof camera displacement. This displacement is utilized to incrementallydisplace the cursor image on a display screen.

The software 68 for determining the camera displacement direction inaccordance with one embodiment of the present invention, shown in FIG.6, begins by computing the absolute value of the difference forsurrounding pixel sample sets from the previous frame. Once it is knownthat the camera has been displaced, for the new frame the samecalculation is undertaken for each of a plurality of sample sets (suchas samples sets B through I), each of which may correspond in size tothe initial sample set A. The sample set (such as the samples sets Bthrough I from the previous frame) whose computed value most closelycorresponds to the value of the set A in the next or current frame isjudged to be the displaced position of the image now recorded as set A.A vector is generated which indicates the direction of movement of thecursor. Thus, if the displaced position in the previous frame is theblock C, the cursor is displaced downwardly and so on.

While a simple example has been used which uses nine fixed blocks Bthrough I and compares them with respect to a block A, those skilled inthe art will appreciate that a number of other variations are possible.For example, instead of using nine spaced blocks within a central regionof the frame, any number of blocks may be iteratively compared andgenerated. Thus, software may be utilized to generate any desired numberof blocks to reduce the granularity of the cursor movements.

In addition, instead of using only a central region of the frame, theentire region of the frame may be scanned for similarities. Similarly,instead of using blocks of fixed size, blocks of variable size may beutilized.

In still other embodiments, algorithms may be utilized to speed thecomparison process. For example, in one embodiment, a comparison may bemade with between two blocks and a value determined. When the next blockexceeds the value already determined for the first block, thecalculation is stopped and that comparison moves on to the next block.Other algorithms are known by those skilled in the art to speedcomparison processes of this type, reducing the number of calculationsand therefore the calculation time.

In some embodiments of the invention, the cursor image is always movedby the same finite amount and the only thing that changes is itsdirection of movement. Since this finite amount is generally a verysmall movement (for example, corresponding to a displacement occurringin {fraction (1/30)} of a second), the cursor image generally followsthe correct direction of camera movement. The associated processor-basedsystem calculates the successive vector displacements.

For the next frame, the analysis may again compare the new center sampleset A with the previous frames surrounding sample sets, such as thesample sets B-I. A new vector is again determined. In one embodiment,the cursor image may always move a fixed amount (if motion is detected)and it is only the direction of the cursor movement that changes.

The system 10 may be implemented by a processor 114 which may becontained within the housing 12. The processor 114 may be coupled to aninterface 116 which may be, for example, a bridge or chipset. Theinterface 116 couples the memory 118 and a display controller 122. Thedisplay controller 122 is coupled to the display 14.

The interface 116 is also coupled to a bus 126 which is in turn coupledto interfaces 128 and 132. The interface 128 couples the camera 16. Theinterface 132 may be coupled to a storage device such as a hard diskdrive 134 which stores the software 30, 44 and 68.

A bus 136, coupled to the interface 132, is also coupled to a serialinput/output (SIO) device 138 and a basic input/output system (BIOS)144. The keyboard 13 may be coupled through the SIO 138.

While the present invention has been described with respect to a limitednumber of embodiments, those skilled in the art will appreciate numerousmodifications and variations therefrom. It is intended that the appendedclaims cover all such modifications and variations as fall within thetrue spirit and scope of this present invention.

What is claimed is:
 1. A method of controlling a processor-based systemcomprising: receiving a current video frame from a camera coupled tosaid system; comparing the current video frame to an immediatelypreceding video frame; and if the difference between the current videoframe and the immediately preceding video frame exceeds a predeterminednoise threshold, then generating a cursor control signal responsive todisplacement of the camera.
 2. The method of claim 1 including receivinga mouse click signal provided by the user through the camera.
 3. Themethod of claim 1 including translating changes in the orientation ofthe camera along two transverse axes.
 4. The method of claim 1 furtherincluding determining whether the orientation of the camera has changedby analyzing the video information.
 5. The method of claim 4 includinganalyzing less than all the pixels captured by the camera to determinewhether the orientation of the camera has changed.
 6. The method ofclaim 1 including linking camera displacement in a given direction todisplacement in the same direction of a cursor.
 7. The method of claim 6including translating the extent of movement of the camera with theextent of movement of the cursor.
 8. An article comprising a medium forstoring instructions that cause a processor-based system to: receive acurrent video frame from a camera coupled to the system; compare thecurrent video frame to an immediately preceding video frame to determinea difference value; and if the difference value exceeds a predeterminednoise threshold, then generate a cursor control signal responsive todisplacement of the camera.
 9. The article of claim 8 further storinginstructions that cause a processor-based system to receive a mouseclick signal provided by the user through the camera.
 10. The article ofclaim 8 further storing instructions that translate changes in theorientation of the camera along two transverse axes.
 11. The article ofclaim 8 further storing instructions that cause a processor-based systemto determine whether the camera orientation is changed by analyzing thevideo information.
 12. The article of claim 11 further storinginstructions that cause a processor-based system to analyze less thanall the pixels captured by the camera to determine whether the cameraorientation has changed.
 13. The article of claim 8 further storinginstructions that cause a processor-based system to link cameradisplacement in a given direction to displacement in the same directionof the cursor.
 14. The antic of claim 13 further storing instructionsthat cause a processor-based system to correlate the extent of movementof the camera with the extent of movement of the cursor.
 15. Aprocessor-based system comprising: a processor; a storage coupled tosaid processor; a camera coupled to said processor; and software storedon said storage to generate a difference value derived, in part, from acomparison of a current video frame to an immediately preceding videoframe to generate a cursor control signal responsive to displacement ofsaid camera.
 16. The system of claim 15 wherein said camera includes apushbutton, said pushbutton operating as a mouse select button.
 17. Thesystem of claim 15 wherein said camera includes a base and a lens, andsaid camera lens is pivotable in three dimensions about said base. 18.The system of claim 15 wherein said cursor control signal is generatedas a result of the analysis of video information captured by saidcamera.
 19. The system of claim 18 wherein said software analyzes lessthan all of the pixels captioned by said camera.
 20. The system of claim18 wherein the extent of movement of said camera controls the extent ofcursor movement.